Method and system for intelligent qualitative and quantitative analysis of digital radiography softcopy reading

ABSTRACT

The present invention describes a method and system for intelligent diagnostic relevant information processing and analysis. Information associated with a patient is processed via an image reading platform. Based on such processed information, a matrix of diagnosis decisions containing diagnostic related information is generated via a matrix of diagnosis decision platform. A diagnostic decision is made based on the diagnostic relevant information. The image reading platform and/or the matrix of diagnosis decision platform encapsulate information and toolkits to be used to manipulate the information.

CROSS-REFERENCE TO RELATED APPLICATIONS

The current Application is based on U.S. Provisional Application No.60/537,558 filed Jan. 21, 2004, and U.S. Provisional Application No.60/562,260 filed Apr. 15, 2004, the entire contents of whichapplications are hereby incorporated by reference.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The invention disclosed herein relates to a system and method forassisted medicine. Particularly, it relates to diagnostic informationanalysis.

2. Description of Related Art

Most radiographic images are complex due to the fact that threedimensional anatomical structures are projected on a two dimensionalimage plane. For example on chest radiographic images, over 60 percentof the lung region may be occluded by ribs. Object(s) of interest suchas nodules may therefore overlap with anatomical structures such asribs, reside in shadows, or may be occluded by other types of objects.These may cause difficulty to observe the object(s) of interest anddiscern the boundary of such object(s). Existing systems have someshared shortcomings or weaknesses in assisting and facilitatingphysicians' softcopy reading of digital/digitized radiographic images.First, most of the existing systems are not capable of providingquantitative measurements, which are often used by physicians to reach adiagnostic decision. This incapability is often related to thedifficulties in segmenting out nodules and/or lesions in images whenstructural/anatomic noise exists due to, for example, the difficultiesstated above. Second, existing systems are not capable of complying withan existing clinical workflow and provide only assistance in certainstages of such a workflow. Third, existing systems usually employblack-box approaches so that it is not possible for physicians tointeract in real time with such systems. As a consequence, such systemscan provide only assistance based on prior knowledge that is built inthe system rather than offering assistance based on physician-specificknowledge and experience.

BRIEF DESCRIPTION OF THE DRAWINGS

The present invention is further described in terms of exemplaryembodiments, which will be described in detail with reference to thedrawings. These drawings are non-limiting exemplary embodiments, inwhich like reference numerals represent similar parts throughout theseveral views of the drawings, and wherein:

FIG. 1( a) illustrates an exemplary clinical workflow for examining aradiographic image;

FIG. 1( b) depicts an exemplary encapsulated structure of the discloseddigital/digitized radiograph softcopy reading system;

FIG. 2( a) shows an exemplary GUI displayed on a portrait monitor;

FIG. 2( b) shows an enlarged picture of a tab controller;

FIG. 2( c) shows an exemplary GUI displayed on a normal monitor;

FIG. 3( a) shows an original image with an arrow pointing at a nodule;

FIG. 3( b) shows an image with nodule-specific image enhancement;

FIG. 3( c) is an example of presenting automatic lung nodule detectionresults by highlighting the suspicious regions;

FIG. 3( d) is an example of applying nodule-specific image enhancementto an automatically detected suspicious nodule region highlighted by thecomputer;

FIG. 4( a) is an exemplary GUI allowing concurrent diagnosis operations;

FIG. 4( b) shows an example of a ROI with a mark indicating a nodule;

FIG. 5( a) shows an exemplary Matrix of Diagnosis Decision (MDD)Platform displayed on a portrait monitor;

FIG. 5( b) shows an exemplary Matrix of Diagnosis Decision (MDD)Platform displayed on a normal monitor;

FIG. 6 shows an exemplary Diagnosis Relevant Information Card;

FIG. 7 shows an example of an encapsulated Diagnostic Information Table;

FIG. 8 illustrates an exemplary embedded consistency check duringinteractive nodule segmentation;

FIG. 9( a) shows an exemplary Clinical Reporting Platform displayed on aportrait monitor;

FIG. 9( b) shows an exemplary Clinical Reporting Platform displayed on anormal monitor;

FIG. 10( a) shows an exemplary GUI with a pop-up dialog box thatinstructs a user to generate a report;

FIG. 10( b) shows an exemplary GUI with a pop-up dialog that requires auser to confirm the inclusion of a specific nodule in a clinical report;

FIG. 10( c) is an exemplary dialog box that instructs a user withrespect to generating a report;

FIG. 10( d) is an exemplary dialog box that requires a user to confirmthe inclusion of a specific nodule in a clinical report;

FIG. 11( a) is a flowchart of an exemplary process for identifyingnodule candidates;

FIG. 11( b) is a flowchart of an exemplary process for removing falsepositive nodule candidates;

FIG. 11( c) is a flowchart of an exemplary process for removing falsepositive nodule candidates using Spider techniques;

FIG. 12 shows exemplary net of insects;

FIG. 13 shows an exemplary surviving spider during nodule candidateidentification;

FIG. 14( a) shows an original region of interest in which a noduleattaches to bones;

FIG. 14( b) illustrates a series of extracted objects corresponding to anodule;

FIG. 15( a) illustrates an object extracted during the removal of falsepositive nodule candidates using Spider techniques;

FIG. 15( b) shows an exemplary template that best captures a targetnodule;

FIG. 16 shows an exemplary block diagram of Spider technique applied tonodule segmentation;

FIG. 17( a) illustrates two ROIs containing nodules; and

FIG. 17( b) illustrates examples of nodules segmented using Spidertechnique.

DETAILED DESCRIPTION OF A PREFERRED EMBODIMENT OF THE INVENTION

The processing described below may be performed by a properly programmedgeneral-purpose computer alone or in connection with a special purposecomputer. Such processing may be performed by a single platform or by adistributed processing platform. In addition, such processing andfunctionality can be implemented in the form of special purpose hardwareor in the form of software or firmware being run by a general-purpose ornetwork processor. Data handled in such processing or created as aresult of such processing can be stored in any memory as is conventionalin the art. By way of example, such data may be stored in a temporarymemory, such as in the RAM of a given computer system or subsystem. Inaddition, or in the alternative, such data may be stored in longer-termstorage devices, for example, magnetic disks, rewritable optical disks,and so on. For purposes of the disclosure herein, computer-readablemedia may comprise any form of data storage mechanism, including suchexisting memory technologies as well as hardware or circuitrepresentations of such structures and of such data.

This invention discloses systems and methods that facilitate integratedplatform(s) capable of facilitating diagnosis information extraction andanalysis to support diagnostic decision making. In some embodiments, thedisclosed invention is applied to digital/digitized radiographic imagesoftcopy reading. Assistant functions may be provided in real timeinteractive fashion so that the assistant functions may be embedded inan optimal workflow. Functionalities facilitating digital/digitizedradiographic image softcopy reading may include, for instance, imagedisplaying, disease-specific enhanced image viewing, annotating,automatic nodule detection, real-time interactive nodule detection andsegmentation, automatic structural clinical reporting, etc. Byencapsulating diagnostic information of high dimension into multipleassistant tools, and organizing such assistant tools to form multiplediagnosis scenario platforms, the disclosed system and method assist auser in reaching a medical diagnosis decision in a manner consistentwith clinical practice workflow. The disclosed system may be used fordifferent purposes, including medical and non-medical. For example, itmay be used as a marking tool in an educational system in full orpartial functions.

In some embodiments, the disclosed invention facilitates a plurality offeatures, such as one or more platform(s) and/or mechanism to supportsoftcopy reading of digital/digitized radiographic images in a mannerconsistent with existing clinical workflow, an open system architecturehaving a diagnostic-information-oriented encapsulated hierarchy,assistance tools that allow a user to interact with the system in realtime, and new algorithms enabling assistance to be rendered for medicaldiagnosis.

In some embodiments, the system and method may be utilized in a mannerconsistent with existing physicians' diagnosis workflow, includingreading images to identify suspicious lesions/nodules, decision makingbased on qualitative and/or quantitative examination andcharacterization, and/or clinical report generation. For example, thedisclosed system may support different groups of functionalities viadistinct platforms such as an Image Reading Platform, a Matrix ofDiagnosis Decision Platform, and a Clinical Reporting Platform.Intelligent assistant toolkits may be provided in a real-time andinteractive manner to facilitate physicians in manipulating the systemcomponents in a manner consistent with their own working styles.

In some embodiments, the system and method may have an open architecturewith a diagnostic-information-oriented encapsulated hierarchy, in whichdiagnostic information of different types at different levels may beencapsulated in appropriate toolkits. Such hierarchical and encapsulatedarchitecture may make system expansion feasible to, for example, handleemerging information as modern technologies develop rapidly.Encapsulated packages containing both data and tools may be deliveredacross different diagnostic workstations, either locally or remotely, sothat users at different locations may deploy such tools to access dataencapsulated in a delivered package.

In some embodiments, the system and method may provide automaticanalysis means in a real-time interactive manner to aid users insoftcopy examination of patient images. Some of the automatic analysismethods performed in a real-time interactive manner may includeinteractive object segmentation and interactive object detection. Thesystem may be open or transparent to users and may allow objectivequantitative analysis performed by the system to be integrated with aphysician's specific knowledge and/or experience to, for instance,improve performance in reaching diagnostic decisions.

In some embodiments, the system and method may be deployed with aplurality of techniques that enable emulation of a spider in catchingfood so that target lesions may be adaptively captured and automaticallysegmented to aid physicians' qualitative and quantitative analysis.

In some embodiments, the system and method may provide other functions,including intelligent automatic nodule detection on the entire image,intelligent real-time interactive nodule detection, real-timeinteractive intelligent nodule segmentation and quantification,real-time manual nodule segmentation and measurement, nodule-specificimage enhancement, automatic clinical report generation. Those exemplaryfunctions may be applied to lung nodules. Each of the exemplaryfunctions is described below.

In some embodiments, the intelligent automatic nodule detection on theentire image may be activated or triggered through a single mouse clickon a button or from a menu. Upon being activated, the functionalcomponent corresponding to the function may automatically identify atarget region such as a lung region and scan the region for eachsuspicious area that possibly contains a lesion. Such automaticdetection may be carried out concurrently with a user's (e.g., aphysician's) manual and/or interactive examination on the same studieswith additional tools that may reside on a same workstation or adifferent workstation located remotely.

In some embodiments, a user may interact with the automatic noduledetection component so that wherever the user points at a specificregion in an image, the system may provide, in real-time, its automaticexamination decision as to whether the specific region indicatedcorresponds to a nodule or not. In addition, the system may also providea confidence measure with its decision indicating a level of confidencewith respect to the decision.

In some embodiments, a user may not be required to trace the boundary ofa nodule in order to derive a segmentation of an underlying nodule.Instead, the user may draw a rectangle around the nodule and the systemmay then automatically extract the boundary of the nodule. The systemmay also automatically compute clinically meaningful features withrespect to such segmented nodule. Examples of such meaningful featuresinclude measurements characterizing the segmented nodule that may beimportant or helpful in assisting a user to make a diagnostic decision.Such measurements may include the size, shape, smoothness of thenodule's boundary and the intensity distribution within the nodule. Insome embodiments, a user may be provided with an option to manuallysegment a nodule and make measurements. In other embodiments, a user mayperform some manual measurement and the system may automatically computeother features accordingly.

In some embodiments, the nodule-specific enhancement may be a real-timeinteractive assistant tool. In some embodiments, the nodule-specificenhancement may be provided for lesion enhancement. The nodule specificenhancement may be applied on the fly to an area to where a user mayhave moved a cursor. Such performed disease-specific enhancement mayprovide a nodule-specific enhanced view of the suspicious region and theenhanced view may be magnified and displayed in a window that isadjustable both in size and in shape.

In some embodiments, the disclosed system and method may allow a user toverify each of identified suspicious nodules that are to be reported. Adetected nodule that is confirmed to be reported may be automaticallyexported, along with its quantitative measurements, to a clinicalreport, which may be read, printed, saved, and reloaded whenever needed.

In some embodiments, the disclosed system and method may automaticallymake appropriate adjustment to its operational parameters to be able toproperly operate in a dynamic environment. For example, depending on adisplay environment, the operational parameters used in displaying agraphical user interface may be automatically adjusted based on, forinstance, the type of monitor used. In another example, font size may beautomatically adjusted according to the resolution of the displaymonitor used. Texts and graphic objects display in the system may alsobe automatically adjusted, e.g., shadow may be automatically added toprovide a better contrast in a displayed image that has a relativelyhigh or a relatively low intensity.

FIG. 1( a) is a flowchart of an exemplary process for softcopy reading.A user may read, at 101, a digital/digitized radiograph image, andidentify, at 102, suspicious regions with or without the assistance of acomputer system. When further examination of a detected suspicious areais considered necessary, at 103, a detailed examination or analysis,either qualitative or quantitative, may be carried out, at 104, tocharacterize the suspicious region. Such characterization may provideevidence for diagnosis. Based on such evidence, a diagnosis decision maybe made and a clinical report may be generated at 105.

In some embodiments, computer assistant toolkits may be grouped andencapsulated in multiple packages so that such tools may be utilized ina manner consistent with an existing clinical workflow. In addition,computer assistant tools may be provided in a real-time and interactivefashion so that they may be embedded in an optimized workflow. Anexemplary embodiment of the encapsulation architecture withcorresponding functions is illustrated in FIG. 1( b). In this exemplaryembodiment, three encapsulated assistant packages may be grouped,including an Image Reading Platform 110 enabling a user to identifysuspicious nodules, a Matrix of Diagnosis Decision (MDD) Platform 120providing a platform where a user may reach a diagnosis decision basedon evidence derived from qualitative/quantitativemeasurements/characterization, and a Clinical Reporting Platform 130enabling generation of summary of information associated with adiagnosis and saving of a diagnostic record. Each of the exemplaryplatforms is described in detail below.

Image Reading Platform

In operation, a user may trigger the Image Reading platform 110 in orderto start softcopy reading. A user may activate any assistant toolencapsulated in this platform or a combination thereof to read an imageassociated with a patient and/or to identify a suspicious region withinthe image. An exemplary display of the Image Reading Platform displayedon a portrait monitor is shown in FIG. 2( a). The exemplary ImageReading Platform comprises a plurality of fields. Examples of includedfields may be, a patient information field 202, a tab controller 204that is accessible in all platforms so that a user may switch back andforth among different diagnostic stages, a display/viewing parameterfield 206, a cursor position and pixel intensity field 208, a toolbarfor interactive assistant tools 210, which may further comprise apatient file selection and open functional icon 211, a window levelsetting adjustment functional icon 212, a functional icon 213 to controlthe display of a user's mark, a functional icon 214 for a batch modeautomatic nodule detection on multiple images, an undo button 215, aredo button 216, a functional icon for automatic nodule detection on acurrent image 217, a functional icon for interactive nodule detection ona current image 218, a nodule-specific image enhancement tool icon 219,a pop-up menu having choices of functions and display settings 220, adisplay window 230 for displaying an image during, e.g., noduledetection, and an interactive detection confidence bar field 240, whichmay pop up when the interactive detection icon 218 is activated. AnImage Reading Platform may be displayed according to automaticallyadjustable display parameters. For example, FIG. 2( c) illustrates adisplay of an Image Reading Platform displayed on a regular monitor. Anenlarged view of the tab controller 204 for switching among differentdiagnosis stages is illustrated in FIG. 2( b).

Based on an Image Reading platform, a user may load a patient image anddisplay such loaded image in the display window 230. Once a patientimage is loaded and displayed, a user may identify a suspicious noduleregion in different operational modes such as in a manual detectionmode, in an automatic detection mode, in an interactive detection mode,or in a combined mode.

In a manual detection mode, a user may identify nodules with or withoutthe help of assistant tools provided in the Image Reading platform. Forexample, a user may specify a suspicious region by manually pointing atthe region via, e.g., a mouse click on a displayed image. When a nodulecandidate is identified, a user may add the detected nodule to aDiagnosis Information Table described below with respect to Matrix ofDiagnosis Decision (MDD) Platform. In some embodiments, a user mayidentify a suspicious region with the help of the system. For example,real-time interactive assistant tool Nodule-specific Image Enhancementtool 219 may be activated to first perform disease-specific imageenhancement which may yield imagery within a region that has enhancedperceptual effect to help the user better understand complex structureswithin the enhanced region. In some embodiments, such enhancement may beapplied on a region of interest (ROI) centered around a dynamic cursorposition. The size of a ROI around a dynamic cursor position may bepredetermined, dynamically computed based on image features, or manuallyadjusted. The shape of such a ROI may be different in differentembodiments. For example, a ROI for enhancement may be circular shape,elliptical shape, or rectangular shape. Magnification may be appliedduring enhancement. In some embodiments, the degree of magnification maybe continuously adjusted by, e.g., dragging the mouse with, e.g., theright button down. FIG. 3( a) shows an example of a part of a chestradiographic image where a nodule is indicated by an arrow. FIG. 3( b)shows the same image with an enhanced region where the enhancement isachieved using an nodule-specific image enhancement tool. In thisexample, the shape of ROI used by the nodule-specific image enhancementtool is a circle.

In some embodiments, automatic nodule detection may be facilitated. Anexample of a nodule may be a pulmonary nodule. Different methods toactivate automatic nodule detection may be implemented. For example,such detection may be triggered via a single mouse click oncorresponding tool icon 214 or through a menu selection. Once thedetection is activated, the system may automatically scan the patientimage to detect for nodules/lesions. Details of nodule detection arediscussed below. If a suspicious nodule structure is identified,information associated with the suspicious nodule structure may bestored for, e.g., additional examination, which may be performedmanually by a user, automatically by a computer system, or interactivelythrough human-machine interaction.

An identified suspicious nodule may be presented or displayed viadifferent means. In some embodiments, a mark may be displayed nearby thedetected suspicious structure pointing at a suspicious nodule area. Insome embodiments, a user may be requested to determine whether theindicated structure corresponds to a likely nodule, whether the detectedsuspicious structure needs further examination, or both. In someembodiments, when either the suspicious nodule is likely to be an actualnodule or the detected nodule may require further examination,information related to the detected nodule candidate may beautomatically added to a Diagnosis Information Table. Details related tothe Diagnosis Information Table are discussed below in describing Matrixof Diagnosis Decision Platform.

In some embodiments, a region containing a detected nodule/lesion may behighlighted to provide an enhanced viewing effect. Highlighting thesuspicious ROI may serve to catch a user's attention. The highlightingmay be achieved via different schemes that differentiate the intensitylevels of the suspicious region and the rest of the image. For example,it may be carried out by increasing the intensity contrast displayedwithin a suspicious region while keeping the intensity contrast of therest of the image the same. Alternatively, this may be achieved bydecreasing the intensity contrast displayed in the rest of the imagewhile keeping the intensity contrast of a suspicious region the same. Asanother alternative, this may also be achieved by simultaneouslyincreasing the display intensity contrast of a suspicious region anddecreasing the display intensity contrast in rest of the image. Thehighlighting effect may also be achieved by making the intensity levelof the suspicious region lower than that of the rest of the image. Inaddition, given that the leveling of image display in a window may beoptimized by a user, one may also choose to keep the current optimizeddisplay settings for a suspicious region and dim out the rest of theimage so that the suspicious region may visually seem to be highlighted.FIG. 3( c) shows an example display of a nodule that is automaticallyidentified. In this example, an automatically identified suspiciousregion 360 is “highlighted” to catch a user's attention. In someembodiments, a user may utilize a nodule-specific image enhancement toolin combination with a marked view mode or a region-highlighted view modeto locate suspicious structures. FIG. 3( d) illustrates a display inwhich nodule-specific image enhancement is applied to a detectedsuspicious region highlighted.

In some embodiments, more than one detected nodule may be grouped in asingle highlighted region covering them all. This may be adopted whendifferent nodules are close by so that a single highlighted region withhighlighted view may visually avoid a cluttered display. When windowlevel settings are changed (e.g., by a user), a display of a suspiciousregion and the rest of the underlying image may be adjusted accordingly,while the contrast between the suspicious region and the rest of theimage may be kept the same to maintain the “highlighting” effect. Insome implementations, a user may be allowed to freely switch between anormal image viewing mode and a view in which a nodule is indicated. Thescheme of highlighting a region to caution a viewer may also be appliedin other scenarios other than detected nodule candidates. For example,it may be applied to other types of diseases or information of differentmodalities.

In some embodiments, automatic nodule detection may be performed in abatch mode for multiple images pre-selected. For example, a user mayselect multiple patient studies and submit a batch job so that detectionmay be performed on all selected images in a batch to automaticallyidentify nodules contained in such images.

In some embodiments, a user may identify nodules in an interactive mode.In some embodiments, this interactive nodule detection mode may beactivated via a single mouse click on, for example, a corresponding toolicon 216. In this mode, a user may point at a region in an image andthen an automatic nodule detection module may operate in real-time toyield an output indicating whether there is a nodule nearby thatparticular location/region or not. Such an output may be provided with aconfidence measure, which may be displayed in different visuallyinformative forms such as a bar or a pie, 220. The confidence measuremay provide a user a reference with respect to a diagnosis decision asto whether the current structure near the indicated region correspondsto a nodule or not. A suspicious region may be saved for furtherexaminations. In some embodiments, nodule detection may also beperformed in an operational mode that is a combination of the abovethree described detection modes. Other assisted tools available from theImage Reading Platform may also be activated in connection with noduledetection.

Some operations that can be activated may be time consuming. In someembodiments, to meet speed requirements in clinical practice and/or toimprove clinical throughput, the operation(s) performed under any of theplatforms may be optimized. For example, processes may be performedsimultaneously in front and back ends. In some arrangements, timeconsuming processes are performed in the backend and real-time functionsare performed in the frontend. Time consuming processes may include, forinstance, some information preparation functions or benchmark automaticnodule detection.

In some embodiments, the operational status of a process running in thebackend may be visually indicated through, for example, a display of apie or others. Such a display may be at the same location as theoriginal tool icon. Putting the tool icon and corresponding processingstatus at the same location may make it easier for a user to rememberwhich task is currently in progress. FIG. 2( c) illustrates an exemplaryinterface showing that backend and frontend concurrent processes are inexecution. FIG. 2( c) shows that when a patient image is loaded, afunction runs in the backend that is extracting certain information thatmay be helpful for a physician's interactive analysis of the image whilea processing status is displayed at or near an “Open” icon 211.Alternatively, a user may concurrently perform diagnosis using otherassistant tools based on existing information and before the wholeinformation extraction completes. FIG. 4( a) illustrates another examplewhere interactive detection is running in the frontend and a benchmarkautomatic nodule detection process is running in the backendsimultaneously. In this example, a processing status associated withautomatic nodule detection icon 217 is displayed indicating that theautomatic nodule detection is running in the backend. An interactivenodule detection icon 218 indicates that interactive nodule detection isconcurrently in progress. A confidence bar 240 with a range, forinstance, from 0.0 to 1.0 may indicate a likelihood with regard to thepresence of a nodule within a current object of interest 402. FIG. 4( b)shows an enlarged view of block 402 in FIG. 4( a), which, for example,indicates that a current object of interest being examined by theInteractive Detection may correspond to an actual nodule.

In some embodiments, a time consuming process may be canceled at anytimeby a user. In some embodiments, a progressive indicator may serve as acancel button. A user may click on a progress indicator to terminate anongoing background process.

In some embodiments, different functions may be automatically configuredbased on data flow dependency. For example, a function that takes theoutput of one or more other functions as its input may be automaticallyactivated after those other functions generate their output. As oneexample, when an information preparation function is still in progress,an automatic nodule detection function that carries out its process on apre-processed image is automatically blocked at the frontend until theinformation preparation function running in the backend completes itsoperation(s).

Matrix of Diagnosis Decision (MDD) Platform

In some embodiments, the Matrix of Diagnosis Decision (MDD) Platformfacilitates various functions related to diagnosis related features. Forexample, it may provide a platform where comprehensive diagnosis-relatedinformation may be presented, qualitative and quantitative analysis maybe activated, and/or diagnosis decisions with respect to individualsuspicious nodule(s) identified under the Image Reading Platform may bemade. The MDD Platform may be encapsulated with various types ofinformation, which may include non-visual information and/or visualinformation. Visual information may include patient information,medication history, lab report(s), image data information, and/orgenotype information. Visual information may include image data and/orpathologic images. The MDD platform may also include real-timeinteractive toolkits encapsulated with different diagnostic information.

In some embodiments, non-visual information may be disease-specificand/or patient-specific and such information may be used by a user indiagnostic decision making. For example, patient specific informationmay be used to make a qualitative assessment as to a level of risk forthe patient to have a specific type of disease. Some prior knowledgerelated to a patient, such as key evidence indicating that the patientis at high risk for a specified disease and/or that some importantparameters may be out of normal ranges, may be highlighted whendisplayed to provide a warning signal to users. In addition tonon-visual information, image related information may be analyzed toderive information relevant to diagnostic decision making. Such analysismay be qualitative or quantitative and the analyzed result may bevisualized and further statistically analyzed. In some embodiments, suchdiagnostic related information, either visual or non-visual, may beencapsulated in different interactive real-time toolkits with functionsthat a user may invoke to assist diagnostic operations.

FIG. 5( a) shows an exemplary MDD Platform displayed on a portraitmonitor. FIG. 5( b) shows an exemplary MDD platform displayed on aconventional monitor. The display of the MDD platform may beautomatically adjusted according to the type of monitor used inpractice. In the illustrated exemplary MDD platform, the MDD Platformmay comprise a Diagnosis relevant Information Card 502, and a patientimage display field 507 with associated assistant functions 506. TheDiagnosis Relevant Information Card 502 may provide visual andnon-visual information that may be used to assist diagnosis decisionmaking. Such information may be displayed or invoked in a working areathrough various encapsulated assistant tools. Different types of visualand non-visual information may be selected using the tab controller 504.Patient related information may be viewed and manipulated using varioustools made available through the encapsulated assistant tools in 506. Inthe illustrated exemplary MDD platform, when Diagnosis Information isselected, the corresponding encapsulated assistant tools are activated,which comprises a display area 509 in which a region of interestcontaining a detected nodule is displayed, and associated interactiveassistant functions 511 to 518.

The MDD Platform may provide various diagnosis assistant tools. Forexample, it may provide tools 506 for displaying and visualizing apatient image. Such visual display of patient image data may provide areference based on which a diagnostic decision is made. Based on such adisplayed image, a user may retrieve information associated with thedisplayed image. For example, information retrieved may indicate inwhich lung lobe a suspicious nodule is located, whether a suspiciousnodule is connected to other anatomic structures, such as vessels ordiaphragm, whether there are other abnormalities that may be related toa detected nodule, etc. Such information may be important to a user inreaching a diagnosis. In some embodiments, similar assistant tools asthose described with respect to the Image Reading Platform may beencapsulated with a patient image. For example, it may include aNodule-specific Image Enhancement tool, a tool for nodule candidate markdisplay or hide, a tool for window leveling, or a tool for image zooming(e.g., zoom in or zoom out, etc.).

The Diagnosis Relevant Information Card 502 in the MDD Platform may beencapsulated with different assistant tools. The Diagnosis RelevantInformation Card 502 may provide visual and non-visual information,which may be encapsulated with different manipulation tools such as themeans to generate qualitative and quantitative measurements on suchinformation. An exemplary GUI for the Diagnosis Relevant InformationCard 502 is shown in FIG. 6. In this example, the Diagnosis RelevantInformation Card 502 comprises a plurality of information groups,including, for instance, a patient information table, an imageinformation table, and a diagnosis information table. Contents in thesetables may be dynamically updated or extended.

In some embodiments, the Diagnosis Relevant Information Card 502 may beconfigured to further possess different features. For example, theDiagnosis Relevant Information Card 502 may be an encapsulation in ahigh-dimensional space. It may also be configured so that it may includeas many categories of information as needed and with as many levels ofdiagnosis related information as needed. For example, the DiagnosisRelevant Information Card 502 may be extended to have an additionalcategory of genotype information represented using, for instance, aGenotype Information Table. In addition, each table may also beextended. For example, the Patient Information Table may be extended toinclude a new sub table containing information related to PreviousMedication History.

In some embodiments, an information table in the Diagnosis RelevantInformation Card 502 may be implemented as an encapsulation with bothdiagnostic related information and functions that can be used tomanipulate the corresponding information. Such encapsulation may makediagnosis based on relevant information more effective. For example, ifa Patient Information Table is selected, other types of informationrelated to the selected patient may be automatically retrieved such ascertain statistics associated a particular disease that the patient issuspected to have. An encapsulated tool may then use such retrievedinformation to, for example, further identify key evidence indicatinge.g., that the patient may be at high risk for a disease and highlightthose key parameters that are out of normal ranges to catch attention ofmedical personnel. In another example, a user may utilize anencapsulated tool to interactively adjust a reference range of aparticular parameter. Such operations may be performed within theencapsulated Patient Information Table.

In some embodiments, whenever a table is selected, its correspondingencapsulated assisted tools may be made available. For example, when adiagnostic information table (see FIG. 6) is selected, activationbuttons corresponding to tools encapsulated with the information in theselected table (e.g., tools that assist qualitative and quantitativemeasuring of suspicious nodules in an image) may be displayed, forexample, underneath the selected table itself. FIG. 7 illustrates suchan example, in which a diagnosis information table is selected andvarious activation buttons (e.g., in the form of icons) associated withencapsulated assisted tools for manipulating information in the selectedtable may be displayed below the table itself. In this example, thediagnostic information table 702 includes different quantitativemeasurements made with respect to a nodule detected and marked in aregion of interest (or an area suspected to have a nodule) as displayedin 708. There are two exemplary toolbars displayed that are associatedwith different types of information. A toolbar 704 corresponds to toolactivation icons associated with the selected diagnostic information orDiagnosis Information Table controllers and a toolbar 706 corresponds totool activation icons for encapsulated real-time assistant toolsassociated with the image displayed in region 708 for diagnosticinformation analysis. The display area 708 may also provide an area inwhich manual, interactive and automatic nodule detection andqualification operations may be carried out. In some embodiments, whenan alternative information diagnosis is selected, differentcorresponding toolkits encapsulated with the selected information may beaccordingly presented.

As one may see illustrated in FIG. 5( a), a user may also select aspecific nodule candidate for examination. A selected nodule may bedisplayed in 509. When the diagnosis information table is activated anda specific nodule candidate is selected for detailed examination, one ormore assistant tools may be used to aid qualitative and quantitativeanalysis on the nodule. For example, such tools may include, but are notlimited to, a tool 511 for window level adjustment of a subimagedisplayed in 509 to yield a better visual effect to support nodulesegmentation, a tool for hiding or displaying a mark at thecorresponding nodule position and/or hiding or displaying the extractednodule boundary or a ruler measurement on the ROI display 509, a ruler513 for measuring the width of a nodule displayed in 509, a ruler 514for measuring the height of a nodule displayed in 509, a tool 515 forperforming manual nodule segmentation to the nodule displayed in 509, atool 516, for performing real-time interactive/automatic nodulesegmentation to the nodule displayed in 509, and a tool 517 to displayhistogram information of the extracted nodule or the region of interestdisplayed in 509, and a tool 518 for help on use of the tools 511 to517, etc.

The Real-time Interactive/automatic Nodule Segmentation function 516 isa nodule segmentation and assessing tool. A user may activate it tosegment a suspicious nodule region by drawing a reference rectanglearound the nodule on a sub-image. The computer may instantaneouslysegment out a nodule and display the segmented result by overlaying thederived nodule boundary on the original sub-image. In some embodiments,some quantitative measures such as size, shape, smoothness of theboundary, and calcification distribution, etc., may be automaticallycalculated in real time and displayed in the diagnosis informationtable. A user may then make a diagnosis decision based on an assessmentof such results. FIGS. 5( a), 5(b), and 7 illustrate a nodule boundarysuperimposed on an image as well as the quantitative measurements madebased on the detected nodule boundary. In some embodiments, means forsegmentation result correction and/or manual segmentation may also beactivated to improve segmentation results yielded by the real-timeinteractive segmentation tools.

In some embodiments, different mechanisms may be implemented in thereal-time interactive nodule segmentation and manual nodule segmentationtools to assure segmentation consistency and quality. For example, whena user draws a nodule boundary to manually segment a nodule or draw areference box around a nodule to let the computer segment a nodule inreal time (e.g., on the subimage displayed in 708), it may beautomatically checked to see whether the drawn boundary or the referencebox actually contains a corresponding nodule position recorded in thediagnostic information table. In some embodiments, when there is norecorded nodule within the boundary or reference box, a warning messagemay be provided. In other embodiments, tools may be provided to help auser locate a marked nodule position and identify the nodule boundary.For example, such a tool may include a window level adjustment 511 toenable display of a subimage for which segmentation is performed to havea better visual effect. Another example of such a tool is 512 thatenables hiding or displaying a mark at a corresponding nodule positionand hiding or displaying the extracted nodule boundary on the ROI image.

It is known that boundaries of a nodule marked by a physician atdifferent times may vary. For example, in segmenting a nodule with asize around 5 mm, a small hand shake may cause substantial differences.The variation may be even bigger when markings are made by differentphysicians. In some embodiments, to reduce inconsistency among differentmarkings made to the same nodule, a user may interact with the system toexamine a segmentation result. In other embodiments, some automatedexamination may be imposed. In some embodiments, in using a real-timeinteractive/automatic nodule segmentation tool to draw a referencerectangle with respect to a nodule, the currently drawn reference boxmay be compared with another reference rectangle that is previouslyconfirmed in terms of position, size, and shape. If the currently drawnreference rectangle substantially deviates from the previous one, somefurther action may be taken to caution a user. For example, a dialog boxmay pop up, warning of the discrepancy and asking a user to make achoice. Through this mechanism, a user is informed of any inconsistency.Such warning may help improve the quality of the segmentation andultimately the diagnosis. FIG. 8 shows an example of such a consistencyassurance mechanism. In this example, the dashed rectangle 806represents a previously confirmed reference box and the solid rectangle804 represents a currently drawn reference box. A dialog box 802 ispopped up to warn a user that inconsistency between the two referenceboxes has been identified after such inconsistency is automaticallydetermined based on information associated with the two boxes. Thedialog box 802 may prompt a user to make a choice between the tworeference boxes. Such a choice may be made based on a user'domain-specific knowledge and/or patient-specific information.

In the example of FIGS. 5( a) and 5(b), there may be other assistanttools that may be encapsulated in different information tables of theDiagnosis Relevant Information Card. Such assistant tools may includetools for information fusion, tools for different informationpresentation (e.g., presentation using numbers, text and/or intuitivegraphs), tools for information adaptation with respect to a user'sspecific knowledge and dynamic configuration, and tools for abnormalitycharacterizing based upon images.

A user may selectively utilize the information and assistant analysistools thereof offered by the MDD Platform. A user may also selectivelyuse information of specific dimension(s) by examining part of theinformation encapsulated. In some situations, a user may check aparticular aspect of the information across time line. In somesituations, a user may compare a particular type of informationcontained in the MDD platform with statistics obtained from elsewhere(e.g., the Internet) for diagnosis purposes. Information and theanalysis thereof may be accessed and performed based on needs so thatthroughput may be improved. Since the wealth of information isencapsulated and made easily accessible, it helps to improve the qualityof diagnosis.

Clinical Reporting Platform

In operation, a user may have control of the workflow of the underlyingsystem. Such control may be partial or full. With an adequate control, auser may utilize the system as a means to assist making a diagnosticdecision. In addition to information and tools provided to assistdiagnosis, other functions may also be provided. One example is tofacilitate verification or confirmation processes for the nodulesdetected. As another example, the system may provide tools to produce aclinical report based on diagnostic activities and results. Differentexemplary embodiments are shown in FIGS. 10( a)-10(d). In FIG. 10( a), auser is prompted via a pop-up dialog box 1002, after diagnosis iscompleted and prior to actually reporting the diagnosis results, to gothrough all the nodule candidates. In FIG. 10( b), a user is promptedvia a dialog box 1004, for each detected nodule, to indicate whether theunderlying nodule is to be reported. FIGS. 10( c) and 10(d) showenlarged views of the dialog boxes 1002 and 1004. In some embodiments,if information in the diagnosis information table is incomplete, awarning dialog box may be popped up to prompt a user to indicate whetherthe operation is to continue.

In some embodiments, a user may select to automatically generate aclinical report according to recorded diagnostic relevant information.An automatically generated clinical report may comprise various types ofinformation. Some types of the information included in a clinical reportare illustrated in FIGS. 9( a) and 9(b), where the Clinical ReportingPlatform may include an index image 908, general patient information906, an examination summary 912, a treatment suggestion field 910 whichmay be filled by a user, regional images of abnormalities (ROIs) withsegmentation results 904, corresponding quantitative measurements andqualitative characterizations 902 for the detected abnormalities, a namefield 914 with a user's (e.g., a physician's) name, as well as a timefield 916 with a date and time when the examination was performed. Auser may enter appropriate information in corresponding fields such asthe name of the physician who performs the diagnosis, date and time ofthe diagnosis, and suggestions as to, for example, treatment or furtherexamination in the corresponding fields. The reporting time may also beentered or alternatively be automatically filled in by the underlyingcomputer. Existence of a physician's name and report generating time maybe provided as a measure of quality. The structure of such a generatedreport may be realized in a variety of different ways. For example, itmay be realized as an XML document, a PDF document, a WORD document, aDICOM structured report, etc. A generated report may also be printed,stored, and loaded in the future.

In some embodiments, additional measures may be deployed to furtherassure the quality and/or security of a clinical report. For example, aclinical report may be encrypted to ensure privacy. A clinical reportmay also be stored using a certain naming convention so that a reportassociated with a patient is unique not only with respect to the patientbut also with respect to each of the physicians who participated in thediagnosis decision. In some embodiments, reports generated for a patientby a physician at different times may be identified using a timeidentifier. An example of such a naming convention may be “patientname+patient ID+image ID+physician's name+time of thereporting+extension name”. In some embodiments, mechanisms may bedeployed to assure that only clinical reports associate with the currentimage may be loaded in for future review.

In some situations, a clinical report may be generated with respect toeach type of imagery information such as a radiographic image. In othersituations, a clinical report may be generated with respect to eachnodule detected from a particular type of image. Each of such instancesof clinical reports may be uniquely identified with respect to differentphysicians and difference times.

We hereby describe in detail methods running behind and supporting thesystem.

Spider Techniques

In some embodiments, nodule detection is realized using an algorithmthat emulates a spider. In the physical world, a spider builds a webwhich is then utilized to capture insects. In some embodiments, a“Dynamic Live Spider” involves a set of algorithms configured to emulatea spider in the physical world. For example, a target object to becaptured by the “Dynamic Live Spider” may be a nodule in adigital/digitized X-ray radiograph. The algorithms disclosed hereinemulating a spider may be configured to detect or capture the presenceof some defined target objects.

In some embodiments, a process of detecting and characterizing nodulesmay be described using an analogy to a process in nature where spidereggs mature into adult spiders, which then form webs that are used tocatch food. In some embodiments, suspicious lesions may be automaticallydetected. In some situations, non-lesion regions that have similarvisual appearance as a lesion may also be detected. Such detectedregions, including ones containing actual lesions and ones that are not,may be considered as eggs of insects. In some embodiments, upon creatingsuch eggs, an “incubation” process may be initiated, in which the eggsgrow to become insects of possibly different species, each of which mayhave varying shapes and sizes corresponding to different anatomies andabnormalities present in an image. Following this incubation process, anatural selection process may begin, in which only spiders may beallowed to survive and other types of insects may be eliminated. Each ofthe surviving spiders may then be given opportunities to build a web toencompass a region of interest. Along a web, a spider may dynamicallystretch its “sensors” along different threads of the web to capture whatis caught on the web. In other words, evidence encountered along a webdynamically established from a surviving spider in an image may bedetected, processed, and analyzed for diagnostic purposes. Thisstretching process may be initiated either from the center of a weboutward or from the outside of a web inward towards the center of theweb. Different image features and different ways of building webs may beapplied based on application needs. In some embodiments, depending onwhether the spider technique is applied to detect nodules or to segmentnodules, a web may be built via different means and searching evidencealong a web may be inward or outward.

Automatic Nodule Detection

In some embodiments, to automatically detect nodules, the disclosedspider technique may be used to emulate the process in which a livingspider actively catches its target food. In some implementations, inapplying the spider technique, a plurality of operational stages may beinvolved. For example, an initial stage may involve candidate generationand mutation, in which nodule candidates are generated as insect eggs.Such candidates may be localized and classified in a candidate locationclassification stage. Based on the classification results, nodules maybe identified in a false positive removal stage.

In some embodiments, initial nodule candidates may be generated based onanalysis performed on a given image. For instance, analysis may be basedon intensity distribution and shape patterns of detected nodules. Insome situations, the visual features of a nodule may be characterized tohave a local intensity peak with surrounding intensity valleys with anapproximate round shape. Such characteristics may be observed from adigital or digitized radiographic image. FIG. 11( a) is a flowchart ofan exemplary process, in which nodule candidates are identified. In thisexemplary process, contrast of a given image may be enhanced at 1101using, for example, wavelet transforms and manipulations. To suppressimage noise and structural/anatomic noise and enhance nodularstructures, a low-pass filter, such as a Laplacian of Gaussians (LoG),may be applied to a contrast enhanced image at 1102. A net of insectsmay be established by computing a topographic sketch of the image in oneor more directions and then identifying regions, at 1103, that havecrossing points of ridge lines and that are surrounded and separated byvalley lines in the topographic sketch image. Such ridge and valleylines may, upon being put together, resemble a net of insects. Anexample of such a net of insects is illustrated in FIG. 12. In someembodiments, a topographic sketch may be derived along 4 directions:horizontal, vertical, and two diagonal directions. Based on the regionsidentified at 1103, the shapes of such regions may be analyzed and thoseregions that have an approximately round shape and of a proper size maybe selected, at 1104, as initial nodule candidates. Such selectedregions may have shapes similar to a spider. An example of a selectedspider is shown in FIG. 13. Although similar in shape or in otherfeatures, such selected spider candidates may not correspond to actualnodules. This may be due to various reasons. For example, superimposing3D anatomic structures onto a 2D image may produce undesirablestructural noise in an image. In some embodiments, spider candidatesgenerated may need to be further examined or classified.

In some exemplary process for lung nodule detection, nodule candidatesmay be classified into a plurality of categories according to, forexample, information associated with the region where a detected noduleresides and the intensity characteristics of a detected nodule. Forexample, such categories may include a category classified based onintensity homogeneity of a detected nodule; a category classified basedon contrast between a detected nodule and its nearby region; a categoryclassified based on boundary strength of a detected nodule; and anycombination thereof.

In some embodiments, for a nodule candidate in each classified category,another processing may be applied to remove false positive candidates.FIG. 11( b) is a flowchart of an exemplary process, in which falsepositive nodules may be removed, for example, from each region ofinterest.

In this exemplary process, contrast between a nodule candidate and itssurrounding backgrounds may be enhanced at 1122. An exemplaryenhancement technique may be wavelet-based enhancement. Features of thecandidate nodule may be further enhanced. Inhomogeneity of the intensitydistribution in an underlying ROI in which the nodule candidate residesmay be compensated at 1124. In some embodiments, grayscale morphologicaloperations may be deployed for such purposes. The intensity profiles ofan enhanced image in the ROI may be analyzed, at 1126, along, forexample, multiple directions. If the profiles in multiple directionsexhibit a particular distribution such as a Gaussian distribution aroundthe nodule candidate and exhibit a certain degree of similarity,determined at 1128, the underlying nodule candidate may be furtherexamined to see whether it is a false positive candidate at 1130.Otherwise, the nodule candidate may be classified at 1144 to be a falsepositive.

In some embodiments, to identify a false positive candidate, varioustypes of information associated with likely features of a nodule may beutilized. For example, information about the homogeneity, brightnesscontrast, and boundary strength may be used when analyzing the intensityprofiles (at 1126). Expected shape of a corresponding intensity profilemay also be used in determining whether a nodule candidate correspondsto a false positive candidate. For nodule candidates that pass theintensity profile check (at 1128), further examination may be applied,at 1130, to remove false positive nodule candidates. In someimplementations, the Spider technique may be applied to detect andremove false positive candidates. If a nodule candidate is classified asfalse positive, determined at 1140, it is rejected at 1144. Otherwise,it is stored as a detected nodule at 1142. Details related to applyingthe spider technique to identify a false positive candidate (at 1130)are discussed below.

FIG. 11( c) illustrates an exemplary process of using the spidertechnique to remove false positive nodules. In this example, asuspicious nodule structure may be extracted at 1131. In someembodiments, this may be achieved by first performing edge detectionwithin the region of interest to produce edge information. Then aplurality of subregions that correspond to the nodule structure may beextracted via, for example, edge constrained region growing where eachof the region growing process may adopt a different threshold within theconstraint of the detected edge information to obtain a differentsubregion as the growing result. In some embodiments, the growing may beinitiated from an estimated center of a nodule candidate within a regionof interest covering the nodule candidate. Boundaries of the subregions,as descriptors of the corresponding subregions, may form a spider web.This multiple step process may emulate a procedure according to which aspider builds and continuously extends a web. When there is weakintensity contrast between the nodule candidate and its surroundingstructures, an extracted subregion may encompass both the target noduleand surrounding anatomical structures connected therewith. An example ofa pulmonary nodule candidate connected to bones is illustrated in FIG.14( a) where an arrow is pointing at the nodule candidate. In suchdescribed process, the lower and upper intensity thresholds may berelaxed in different scales so that different extraction results usingdifferent sets of threshold values may be derived. The amount ofrelaxation of the lower and upper thresholds at each step may beprefixed or may be dynamically adjusted. FIG. 14( b) illustratesexemplary subregions extracted in this multiple step process. Theycorrespond to the nodule candidate illustrated in FIG. 14( a). In theseexemplary results, the extracted subregions encompass not only thenodule region but also nearby anatomic structures such as bones.

In some embodiments, further analysis may be applied to a nodule regioninstead of an entire extracted subregion 1132. Such a nodule region maybe smaller than an entire subregion. To approximately identify thenodule region, a plurality of templates with various sizes are generatedfor each of the subregions. In some embodiments, each of the templatesmay center around a center of a nodule candidate and overlap with theunderlying subregion. Such an overlap produces or yields an area ofobject of interest. In some embodiments, templates may be round withvarious sizes which may be pre-defined or may be dynamically computed. Atemplate may also have a different shape such as oval with various sizesand orientations which may be pre-defined or dynamically computed. Anarea of object of interest may represent estimates of the nodule region.

In some embodiments, certain features of an object of interest may becomputed at 1133. Such features may include, but not limited to, size,circularity, boundary smoothness, an area measurement (for example, theratio between the area of the object of interest OOI and the area of thetemplate), ratio between the length of the part of the template boundarythat intersects the extracted subregion and the perimeter of thetemplate, edge strength along the boundary of an OOI, the difference ofedge strength between the inner boundary and outer boundaries of an OOI,etc. A template that best captures the underlying nodule may bedetermined through examination of such features. Example of a subregionand a determined best template are illustrated in FIGS. 15( a) and15(b), respectively. FIG. 15( a) shows an example of an extractedsubregion containing both the nodule and bones and FIG. 15( b) shows anexemplary template identified to best capture the nodule using thefeatures computed.

In some embodiments, a decision as to whether a nodule candidate is afalse positive may be determined, at 1134, by analyzing the computedfeatures in connection with utilizing knowledge-based reasoning. Such aprocess may emulate the process of a spider, on a web, sensing itstarget food described by certain features. For example, an actual nodulemay be generally known to have an approximately round/oval shape, have arelatively higher occupation area, have small ratios between the lengthsof the boundaries cutting the OOI from the whole extracted object andthe perimeter of the template, and have relatively high edge strengthalong the boundaries of the OOI. In addition, a category of a nodulecandidate may be utilized in the knowledge-based reasoning. For example,if a nodule shows strong inhomogeneous intensity distribution, it maysuggest that the nodule is overlaid on a rib. Therefore, the effect ofthe rib edge in evaluating the edge strength along the OOI boundary maybe taken into account. In addition to examining candidates in theintensity domain, intensity gradients and edges may also be analyzedalong the web lines, for example, both in longitude and latitudedirections. The features of the nodule candidate may include, but arenot limited to, the magnitude and orientation of edges, theirstatistical distributions along web lines, such as the mean values andstandard deviations, local and global spatial relationships of thestrongest edges along the longitude lines. These features may be dividedinto groups according to their correlation strength and may be used asinput to a set of cascaded classifiers to identify true nodules.

If a candidate is considered to be a nodule during the above reasoningprocess, the underlying candidate may be saved in a nodule list at 1135and presented to the user for further examination. Otherwise, it isrejected as a false positive at 1136.

Lung Nodule Segmentation

In some embodiments, the spider technique may be deployed in nodulesegmentation. In some embodiments, such application of the spidertechnique may be implemented in real time processing. FIG. 16 is aflowchart of an exemplary process of nodule segmentation.

In this exemplary process, for a given nodule location, a spider maybuild a web in an area where the nodule resides. Local image propertiesmay be analyzed at 1602 along the lines of the web. The web may beestablished using different means, including gridding or gridding withdiagonal directions. By establishing a web, 2D processing may be reducedto 1D processing to reduce computational cost. Exemplary imageproperties to be analyzed may include intensity profile of the localimage area, corresponding curvature of the intensity profile, curvatureof a local histogram, edge strength, or a profile of a correspondingLaplacian of Gaussian (LoG) image.

Based on the local image properties, special features representingnodule boundaries may be identified at 1603 along the lines of the web.For example, by analyzing the intensity distribution in an nodule area,it may be recognized that although the intensity contrast along theboundary lines may be vague and the intensity distribution of nodulesmay vary, strong responses may still be generated around the noduleboundaries after certain processing such as applying a Laplacian ofGaussian (LoG) filter combined with edge enhancement filter, findinglocal maxima of the curvature of the local intensity profile, orapplying a combination of both. Those positions where strong responsesare identified may be consider to represent potential boundary positionsof the nodule.

In some embodiments, to make segmentation more reliable and robust withrespect to image noise and/or structural/anatomical noise, boundarypoints may be first roughly identified by finding local maxima on 1-Dintensity profiles of an edge-enhanced and LoG-filtered image. This maymake the segmentation less sensitive to image noise due to the fact thatafter applying an LoG filter, the effect of image noise and structuresother than the nodule may be suppressed. However, the edge-enhanced andLoG-filtered images may be somehow distorted from the original images.Analysis of the local intensity profile curvatures of the original imageand the edge enhanced image may be further applied to fine tune thesegmentation. To do so, small search windows may be applied on 1-Dprofile curvature curves around the boundary points identified from theLoG intensity profiles, and the points of local maxima response withfair edge strength may be considered as the refined nodule boundarypoints.

In some embodiments, a segmented nodule may be outlined to derive itsboundary at 1604, based on the nodule boundary points identified at1603. The outlining operation may be performed based on vertices of apiece-wise-smooth polygon of the nodule boundary. The smoothness of theboundaries may be adjusted by configuring the denseness of the weblines.

In some embodiments, determination of boundary points between verticesmay be made in different ways. For example, local histograms around twoadjacent vertices may be analyzed so that an optimal local intensitythreshold can be chosen. It may also be achieved by an interpolationwhen, for example, vertices on the original boundary polygon are notadequately dense. In some situations, some of the identified boundarypoints may not be on the true boundary positions. In some embodiments,to solve this problem, neighboring boundary points may be utilized torefine, at 1605, the boundary by removing outliers according to, forexample, a certain degree of predefined stiffness. FIG. 17( a) shows twoexample images each of which contain a nodule. FIG. 17( b) showssegmentation results derived from the two images in FIG. 17( a) usingthe spider technique.

Although the foregoing embodiments have been described in some detailfor purposes of clarity of understanding, the invention is not limitedto the details provided. There are many alternative ways of implementingthe invention. The disclosed embodiments are illustrative and notrestrictive.

1-64. (canceled)
 65. A method, comprising: detecting, in an image, anarea that contains one or more candidates of an object of apre-determined type; performing, if the area containing one or morecandidates is detected, analysis to affirm or disaffirm the existence ofthe object of the pre-determined type with respect to each of thecandidates, wherein said detecting is performed in one of a manual mode,an automatic mode, an interactive mode, and a combination thereof, andthe analysis is conducted based on patient-specific and/ordisease-specific information associated with the image.
 66. The methodaccording to claim 65, wherein the image includes a radiographic image.67. The method according to claim 65, wherein the object of apre-determined type includes a nodule.
 68. The method according to claim65, wherein the information associated with the image includes visualand non-visual information that is patient specific and/or diseasespecific or information computed from the image.
 69. The methodaccording to claim 65, wherein said detecting in a manual mode isperformed by a user via an interface capable of facilitating the user tomanually perform the detection.
 70. The method according to claim 65,wherein said detecting in an automatic mode is performed by an automaticdetection mechanism which emulates a spider to detect the area with oneor more object of interest of a pre-determined type.
 71. The methodaccording to claim 70, wherein said detecting in an interactive mode isaccomplished in a process in which an automatic detection is performedwith respect to a region specified by a user.
 72. The method accordingto claim 65, wherein said detecting in an automatic mode comprises:identifying an area in the image that potentially includes one or morecandidates of an object of a pre-determined type when the visual and/ornon-visual information associated with the area satisfy a certaincondition; classifying the one or more candidates into a plurality ofcategories; and removing a candidate that is classified as a falsetarget.
 73. The method according to claim 72, wherein said identifyingan area with one or more candidates comprises: enhancing the image toproduce a first enhanced image; filtering the first enhanced image toproduce a filtered image; computing a topographic sketch based on thefiltered image to produce a topographic image, wherein the topographicsketch is produced in a plurality of directions; determining ridge andvalley lines in the topographic image; locating a region containing oneor more crossing points where a plurality of ridge lines are surroundedand/or separated by a plurality of valley lines; and identifying theregion as a candidate of the pre-determined object type when a geometricfeature associated with the region satisfies a certain condition. 74.The method according to claim 73, wherein the certain condition includesat least one of: a criterion related to a shape of the region; or acriterion related to a size of the region.
 75. The method according toclaim 74, wherein the criterion related to the shape of the regionindicates that the region has a substantially round shape.
 76. Themethod according to claim 74, wherein the criterion related to the sizeof the region indicates that the region has a size falling into apre-defined range.
 77. The method according to claim 72, wherein acategory of a candidate includes at least one of: a category with acertain intensity homogeneity of a detected nodule; a category with acertain degree of contrast between a detected nodule and its nearbyregion; a category with a certain degree of edge strength along theboundary of a detected nodule; and any combination thereof.
 78. Themethod according to claim 72, wherein said removing comprises:generating a first enhanced region of interest around each classifiedcandidate to improve intensity contrast; generating a second enhancedregion of interest based on the first enhanced region of interest toimprove intensity homogeneity; determining whether the region ofinterest represents a false target based on intensity profile analysisof the second enhanced region of interest; and eliminating the region ofinterest if the region of interest is determined to be a false target.79. The method according to claim 78, wherein said determiningcomprises: determining a center in the region of interest; performingedge detection within the region of interest to produce edgeinformation; generating a plurality of subregions in the region ofinterest via edge constrained region growing from the center using aplurality of corresponding thresholds and based on the edge information;generating a plurality of templates for each of the subregions, whereineach of the templates centers around the center and overlaps with theunderlying subregion yielding an object of interest; computing at leastone feature for each object of interest; determining a best templatefrom the plurality of templates based on the at least one featurecomputed for each object of interest wherein the best template capturesan estimated nodule; determining whether the estimated nodule is anobject of the pre-determined type; and classifying the region ofinterest as a false target if each estimated nodule from each subregiondoes not represent an object of the predetermined type.
 80. The methodaccording to claim 79, wherein each of the templates has a circularshape with a different radius around the center.
 81. The methodaccording to claim 79, wherein the at least one feature comprises atleast one of: a size measure of an object of interest; a measure ofcircularity of an object of interest; a measure of boundary smoothnessof an object of interest; an area measure of an object of interest; alength of a portion of the template boundary that intersects theunderlying subregion; a perimeter of a template whose overlap with theunderlying subregion yields the object of interest; a measure indicatingstrength of edge along the boundary of the object of interest; and adifference in edge strength between an inner boundary and an outerboundary of an object of interest.
 82. The method according to claim 65,further comprising reaching a medical decision based on the results fromsaid detecting and said analysis.
 83. The method according to claim 72,wherein the medical decision is a diagnosis decision.
 84. The methodaccording to claim 65, further comprising automatically generating areport based on the results from said analysis and/or the medicaldecision and the user's confirmation.
 85. The method according to claim74, further comprising automatically summarizing the detection andanalysis results in the report.
 86. A method for detecting an object,comprising: determining an initial location in an image; performing edgedetection within the image to produce edge information; generating aplurality of subregions in the image via edge constrained region growingfrom the initial location using a plurality of corresponding thresholdsand based on the edge information; generating a plurality of templatesfor each of the subregions, wherein each of the templates centers aroundthe initial location and overlaps with the underlying subregion yieldingan area of object of interest; computing at least one feature for eacharea of object of interest, and; selecting a best template from theplurality of templates based on the at least one feature computed foreach area of object of interest.
 87. The method according to claim 86,wherein each of the templates has a circular shape with a differentradius around the center.
 88. The method according to claim 86, whereinthe at least one feature comprises at least one of: a size measure of anobject of interest; a measure of circularity of an object of interest; ameasure of boundary smoothness of an object of interest; an area measureof an object of interest; a length of a portion of the template boundarythat intersects the underlying subregion; a perimeter of a templatewhose overlap with the underlying subregion yields the object ofinterest; a measure indicating strength of edge along the boundary ofthe object of interest; and a difference in edge strength between aninner boundary and an outer boundary of an object of interest.
 89. Themethod according to claim 86, further comprising identifying whether thearea of object of interest within the best template is an object of apre-determined type.
 90. The method according to claim 89, wherein saididentifying comprises: computing a feature along a pre-defined directionwithin the area of object of interest; and determining whether the areaof object of interest represents an object of the pre-determined typebased on the computed feature.
 91. A method of detecting an objectcandidate, comprising: computing a topographic sketch in an image toproduce a topographic image, wherein the topographic sketch is producedin a plurality of directions; determining ridge and valley lines in thetopographic image; locating a region in the image containing one or morecrossing points where a plurality of ridge lines are surrounded and/orseparated by a plurality of valley lines; identifying the region as acandidate of a pre-determined object type when at least one featurecomputed from the region satisfies a certain condition.
 92. The methodaccording to claim 91, wherein the certain condition includes at leastone of: a criterion related to a shape of the region; or a criterionrelated to a size of the region.
 93. The method according to claim 92,wherein the criterion related to the shape of the region indicates thatthe region has a substantially round shape.
 94. The method according toclaim 92, wherein the criterion related to the size of the regionindicates that the region has a size falling into a pre-defined range.95. The method according to claim 91, wherein the image is an enhancedimage.
 96. The method according to claim 91, wherein the image is afiltered image. 97-111. (canceled)